Batch Reinforcement Learning on the Industrial Benchmark: First Experiences

The Particle Swarm Optimization Policy (PSO-P) has been recently introduced and proven to produce remarkable results on interacting with academic reinforcement learning benchmarks in an off-policy, batch-based setting. To further investigate the properties and feasibility on real-world applications, this paper investigates PSO-P on the so-called Industrial Benchmark (IB), a novel reinforcement learning (RL) benchmark that aims at being realistic by including a variety of aspects found in industrial applications, like continuous state and action spaces, a high dimensional, partially observable state space, delayed effects, and complex stochasticity. The experimental results of PSO-P on IB are compared to results of closed-form control policies derived from the model-based Recurrent Control Neural Network (RCNN) and the model-free Neural Fitted Q-Iteration (NFQ). Experiments show that PSO-P is not only of interest for academic benchmarks, but also for real-world industrial applications, since it also yielded the best performing policy in our IB setting. Compared to other well established RL techniques, PSO-P produced outstanding results in performance and robustness, requiring only a relatively low amount of effort in finding adequate parameters or making complex design decisions

A Benchmark Environment Motivated by Industrial Control Problems

In the research area of reinforcement learning (RL), frequently novel and promising methods are developed and introduced to the RL community. However, although many researchers are keen to apply their methods on real-world problems, implementing such methods in real industry environments often is a frustrating and tedious process. Generally, academic research groups have only limited access to real industrial data and applications. For this reason, new methods are usually developed, evaluated and compared by using artificial software benchmarks. On one hand, these benchmarks are designed to provide interpretable RL training scenarios and detailed insight into the learning process of the method on hand. On the other hand, they usually do not share much similarity with industrial real-world applications. For this reason we used our industry experience to design a benchmark which bridges the gap between freely available, documented, and motivated artificial benchmarks and properties of real industrial problems. The resulting industrial benchmark (IB) has been made publicly available to the RL community by publishing its Java and Python code, including an OpenAI Gym wrapper, on Github. In this paper we motivate and describe in detail the IB's dynamics and identify prototypic experimental settings that capture common situations in real-world industry control problems

A Benchmark Environment Motivated by Industrial Control Problems

In the research area of reinforcement learning (RL), frequently novel and promising methods are developed and introduced to the RL community. However, although many researchers are keen to apply their methods on real-world problems, implementing such methods in real industry environments often is a frustrating and tedious process. Generally, academic research groups have only limited access to real industrial data and applications. For this reason, new methods are usually developed, evaluated and compared by using artificial software benchmarks. On one hand, these benchmarks are designed to provide interpretable RL training scenarios and detailed insight into the learning process of the method on hand. On the other hand, they usually do not share much similarity with industrial real-world applications. For this reason we used our industry experience to design a benchmark which bridges the gap between freely available, documented, and motivated artificial benchmarks and properties of real industrial problems. The resulting industrial benchmark (IB) has been made publicly available to the RL community by publishing its Java and Python code, including an OpenAI Gym wrapper, on Github. In this paper we motivate and describe in detail the IB's dynamics and identify prototypic experimental settings that capture common situations in real-world industry control problems

Intrafractional dose variation and beam configuration in carbon ion radiotherapy for esophageal cancer

Background: In carbon ion radiotherapy (CIR) for esophageal cancer, organ and target motion is a major challenge for treatment planning due to potential range deviations. This study intends to analyze the impact of intrafractional variations on dosimetric parameters and to identify favourable settings for robust treatment plans. Methods: We contoured esophageal boost volumes in different organ localizations for four patients and calculated CIR-plans with 13 different beam geometries on a free-breathing CT. Forward calculation of these plans was performed on 4D-CT datasets representing seven different phases of the breathing cycle. Plan quality was assessed for each patient and beam configuration. Results: Target volume coverage was adequate for all settings in the baseline CIR-plans (V95 > 98% for two-beam geometries, > 94% for one-beam geometries), but reduced on 4D-CT plans (V95 range 50–95%). Sparing of the organs at risk (OAR) was adequate, but range deviations during the breathing cycle partly caused critical, maximum doses to spinal cord up to 3.5x higher than expected. There was at least one beam configuration for each patient with appropriate plan quality. Conclusions: Despite intrafractional motion, CIR for esophageal cancer is possible with robust treatment plans when an individually optimized beam setup is selected depending on tumor size and localization

Esthesioneuroblastoma in pediatric and adolescent age. A report from the TREP project in cooperation with the Italian Neuroblastoma and Soft Tissue Sarcoma Committees

<p>Abstract</p> <p>Background</p> <p>Esthesioneuroblastoma (ENB) is a rare, aggressive tumor with no established treatment in children. We analyzed a series of pediatric ENB patients with the aim of improving our knowledge of this disease.</p> <p>Methods</p> <p>9 patients (6 males; age 0.9-18 years, median 9.9) were identified by searching the AIEOP (<it>Italian Association of Pediatric Hematology and Oncology</it>) registry and the national databases of rare tumors, soft tissue sarcomas (STS) and neuroblastomas. The data on the cases included in STS treatment protocols were collected prospectively and histology was centrally reviewed; the data and histology concerning the other children were reviewed for the purpose of this analysis.</p> <p>Results</p> <p>All tumors occurred in the sinonasal region with bone erosion (7 patients) and intracranial (4) or intraorbital (4) extension. Three patients were in Kadish stage B, and 6 in stage C. Complete tumor resection was very difficult to achieve, but adding chemotherapy and radiotherapy enabled tumor control in 8 patients. Response to chemotherapy was evident in 5/7 evaluable cases. Radiotherapy (48.5-60 Gy) was delivered in all children but one, due to early disease progression. With a median follow-up of 13.4 years (range 9.2-22.9), 7 patients are alive in 1^stand one in 2nd complete remission. All surviving patients developed treatment-related sequelae, the most frequent being endocrine dysfunctions (4 patients) and craniofacial growth impairments (4 patients).</p> <p>Conclusions</p> <p>Our findings confirm that ENB in children has an aggressive presentation, but multimodal therapy can cure most patients. Our results are encouraging but future strategies must optimize treatment in terms of survival and related morbidities.</p

Biological in-vivo measurement of dose distribution in patients' lymphocytes by gamma-H2AX immunofluorescence staining: 3D conformal- vs. step-and-shoot IMRT of the prostate gland

<p>Abstract</p> <p>Background</p> <p>Different radiation-techniques in treating local staged prostate cancer differ in their dose- distribution. Physical phantom measurements indicate that for 3D, less healthy tissue is exposed to a relatively higher dose compared to SSIMRT. The purpose is to substantiate a dose distribution in lymphocytes <it>in-vivo </it>and to discuss the possibility of comparing it to the physical model of total body dose distribution.</p> <p>Methods</p> <p>For each technique (3D and SSIMRT), blood was taken from 20 patients before and 10 min after their first fraction of radiotherapy. The isolated leukocytes were fixed 2 hours after radiation. DNA double-strand breaks (DSB) in lymphocytes' nuclei were stained immunocytochemically using the gamma-H2AX protein. Gamma-H2AX foci inside each nucleus were counted in 300 irradiated as well as 50 non-irradiated lymphocytes per patient. In addition, lymphocytes of 5 volunteer subjects were irradiated externally at different doses and processed under same conditions as the patients' lymphocytes in order to generate a calibration-line. This calibration-line assigns dose-value to mean number of gamma-H2AX foci/ nucleus. So the dose distributions in patients' lymphocytes were determined regarding to the gamma-H2AX foci distribution. With this information a cumulative dose-lymphocyte-histogram (DLH) was generated. Visualized distribution of gamma-H2AX foci, correspondingly dose per nucleus, was compared to the technical dose-volume-histogram (DVH), related to the whole body-volume.</p> <p>Results</p> <p>Measured <it>in-vivo </it>(DLH) and according to the physical treatment-planning (DVH), more lymphocytes resulted with low-dose exposure (< 20% of the applied dose) and significantly fewer lymphocytes with middle-dose exposure (30%-60%) during Step-and-Shoot-IMRT, compared to conventional 3D conformal radiotherapy. The high-dose exposure (> 80%) was equal in both radiation techniques. The mean number of gamma-H2AX foci per lymphocyte was 0.49 (3D) and 0.47 (SSIMRT) without significant difference.</p> <p>Conclusions</p> <p><it>In-vivo </it>measurement of the dose distribution within patients' lymphocytes can be performed by detecting gamma-H2AX foci. In case of 3D and SSIMRT, the results of this method correlate with the physical calculated total body dose-distribution, but cannot be interpreted unrestrictedly due to the blood circulation. One possible application of the present method could be in radiation-protection for <it>in-vivo </it>dose estimation after accidental exposure to radiation.</p

Intensity modulated radiotherapy (IMRT) in the treatment of children and Adolescents - a single institution's experience and a review of the literature

<p>Abstract</p> <p>Background</p> <p>While IMRT is widely used in treating complex oncological cases in adults, it is not commonly used in pediatric radiation oncology for a variety of reasons. This report evaluates our 9 year experience using stereotactic-guided, inverse planned intensity-modulated radiotherapy (IMRT) in children and adolescents in the context of the current literature.</p> <p>Methods</p> <p>Between 1999 and 2008 thirty-one children and adolescents with a mean age of 14.2 years (1.5 - 20.5) were treated with IMRT in our department. This heterogeneous group of patients consisted of 20 different tumor entities, with Ewing's sarcoma being the largest (5 patients), followed by juvenile nasopharyngeal fibroma, esthesioneuroblastoma and rhabdomyosarcoma (3 patients each). In addition a review of the available literature reporting on technology, quality, toxicity, outcome and concerns of IMRT was performed.</p> <p>Results</p> <p>With IMRT individualized dose distributions and excellent sparing of organs at risk were obtained in the most challenging cases. This was achieved at the cost of an increased volume of normal tissue receiving low radiation doses. Local control was achieved in 21 patients. 5 patients died due to progressive distant metastases. No severe acute or chronic toxicity was observed.</p> <p>Conclusion</p> <p>IMRT in the treatment of children and adolescents is feasible and was applied safely within the last 9 years at our institution. Several reports in literature show the excellent possibilities of IMRT in selective sparing of organs at risk and achieving local control. In selected cases the quality of IMRT plans increases the therapeutic ratio and outweighs the risk of potentially increased rates of secondary malignancies by the augmented low dose exposure.</p

Treatment of non-small cell lung cancer with intensity-modulated radiation therapy in combination with cetuximab: the NEAR protocol (NCT00115518)

BACKGROUND: Even today, treatment of Stage III NSCLC still poses a serious challenge. So far, surgical resection is the treatment of choice. Patients whose tumour is not resectable or who are unfit to undergo surgery are usually referred to a combined radio-chemotherapy. However, combined radio-chemotherapeutic treatment is also associated with sometimes marked side effects but has been shown to be more efficient than radiation therapy alone. Nevertheless, there is a significant subset of patients whose overall condition does not permit administration of chemotherapy in a combined-modality treatment. It could be demonstrated though, that NSCLCs often exhibit over-expression of EGF-receptors hence providing an excellent target for the monoclonal EGFR-antagonist cetuximab (Erbitux(®)) which has already been shown to be effective in colorectal as well as head-and-neck tumours with comparatively mild side-effects. METHODS/DESIGN: The NEAR trial is a prospective phase II feasibility study combining a monoclonal EGF-receptor antibody with loco-regional irradiation in patients with stage III NSCLC. This trial aims at testing the combination's efficacy and rate of development of distant metastases with an accrual of 30 patients. Patients receive weekly infusions of cetuximab (Erbitux(®)) plus loco-regional radiation therapy as intensity-modulated radiation therapy. After conclusion of radiation treatment patients continue to receive weekly cetuximab for 13 more cycles. DISCUSSION: The primary objective of the NEAR trial is to evaluate toxicities and feasibility of the combined treatment with cetuximab (Erbitux(®)) and IMRT loco-regional irradiation. Secondary objectives are remission rates, 3-year-survival and local/systemic progression-free survival